1. Confusion Matrix

Example 1

Example 2

Type 1 and Type 2 Errors

• Sometimes in the literature, you'll see False Positives and False Negatives as Type 1 and Type 2 errors. Here is the correspondence:

1. Type 1 Error (Error of the first kind, or False Positive): In the medical example, this is when we misdiagnose a healthy patient as sick.
2. Type 2 Error (Error of the second kind, or False Negative): In the medical example, this is when we misdiagnose a sick patient as healthy.

Understanding the Confusion Matrix

1. True Positive (TP) is the number of correct predictions that an example is positive. This means positive class correctly identified as positive. Example: Given class is spam and the classifier has been correctly predicted it as spam.
2. False Negative (FN) is the number of incorrect predictions that an example is negative. This means positive class incorrectly identified as negative. Example: Given class is spam however, the classifier has been incorrectly predicted it as non-spam.
3. False Negative (FN) is the number of incorrect predictions that an example is negative. This means positive class incorrectly identified as negative. Example: Given class is spam however, the classifier has been incorrectly predicted it as non-spam.
4. False Negative (FN) is the number of incorrect predictions that an example is negative. This means positive class incorrectly identified as negative. Example: Given class is spam however, the classifier has been incorrectly predicted it as non-spam.

2. Accuracy

• Accuracy is the proportion of the total number of predictions that are correct. ## $Accuracy = \frac{Correctly \ classified \ points}{All \ points}$

Question: When Accuracy don't work?

Example: Model 1: All transactions are good

Example: Model 2: All transactions are fraudulent

3. False Negatives & False Positives

Question: In the medical example, what is worse, a False Positive, or a False Negative?
Answer: A False Positive implies sending a healthy person to get more tests. This is slightly inconvenient, but ok. A False Negative implies sending a sick person home, which can be disastrous!

Question: In the email example, what is worse, a False Positive, or a False Negative?
Answer: A False Negative implies a spam message will make its way into your inbox. This is slightly inconvenient, but ok. A False Positive implies missing an e-mail from your dear grandma, which can be disastrous!

4. Precision and Recall

A. Precision

• Precision is ratio of total number of correctly classified positive examples and the total number of predicted positive examples. It shows correctness achieved in positive prediction. ## $Precision = \frac{TP}{TP + \ FP}$

B. Recall

• Sensitivity is also referred as True Positive Rate or Recall. It is measure of positive examples labeled as positive by classifier. It should be higher.
• For instance, proportion of emails which are spam among all spam emails. ## $Recall = \frac{TP}{TP + \ FN}$

5. F1 Score

-F1 score is a weighted average of the recall (sensitivity) and precision. F1 score might be good choice when you seek to balance between Precision and Recall.

Question: What averages of recall and precision is not a good idea, but weighted average it?

Example: Model 1: All transactions are good

Example: Model 2: All transactions are fraudulent

F1 score = Harmonic Mean of Precision and Recall

Example: Model 1: All transactions are good

• Notice: F1-Score is lower than the average of Recall and Precision and, hence a better metric.

6. F-beta Score

• The F-beta score is the weighted harmonic mean of precision and recall, reaching its optimal value at 1 and its worst value at 0

Understanding F-beta score

• When beta=1, the F-beta score is equivalent to the F-1 Score.
• When beta=0.5, this score is the F-0.5 score, and so on.

Question: Out of the following three models, which one should have an F-beta score of 2, 1, and 0.5? Match each model with its corresponding score.

1. Detecting malfunctioning parts in a spaceship

• For the spaceship model, we can't really afford any malfunctioning parts, and it's ok if we overcheck some of the parts that are working well. Therefore, this is a high recall model, so we associate it with beta = 2.

1. Sending phone notifications about videos a user may like

• For the notifications model, since it's free to send them, we won't get harmed too much if we send them to more people than we need to. But we also shouldn't overdo it, since it will annoy the users. We also would like to find as many interested users as we can. Thus, this is a model which should have a decent precision and a decent recall. Beta = 1 should work here.

1. Sending promotional material in the mail to potential clients

• For the Promotional Material model, since it costs us to send the material, we really don't want to send it to many people that won't be interested. Thus, this is a high precision model. Thus, beta = 0.5 will work here.

7. Specificity

• Specificity is also know as True Negative Rate. It is measure of negative examples labeled as negative by classifier. There should be high specificity. For instance, proportion of emails which are non-spam among all non-spam emails. ## $Specificity = \frac{TN}{TN + \ FP}$

8. Receiver Operating Characteristic (ROC) Curve

Example:

• Consider a split (say good split) and find TPR and FPR

• We will move the divider around the line and try to find the True Positive Rate and False Positive Rate for every possible split.

• Now, we try to plot this: TPR as a function of FPR
• Then, we calculate the under the curve is called AUC (Area Under the Curve). Here, AUC = 0.8

Question: What happens when the split is Perfect?

• AUC = 1

Question: What happens when the split is Random (Bad)?

• AUC = 0.5

Summarizing

Conclusion:

• Closer the area under the ROC curve to 1, the better the model is.

Question: Can the area under the ROC curve be less than 0.5?

• Yes, it can be all the way to zero. It will have flipping of the data points in case of a perfect split

9. Regression Metrics

A. Mean Absolute Error

• MAE is calculated by taking the sum of the absolute value of error.
• Error = Real output minus Predicted output
• MAE has a problem that it is not differentiable, it would be problematic if we want to use methods such as Gradient Descent

B. Mean Squared Error

• MSE is calculated by the sum of square of prediction error which is real output minus predicted output and then divide by the number of data points. It gives you an absolute number on how much your predicted results deviate from the actual number.

C. RMSE

• Root Mean Square Error(RMSE) is the square root of MSE. It is used more commonly than MSE because:
• Firstly sometimes MSE value can be too big to compare easily.
• Secondly, MSE is calculated by the square of error, and thus square root brings it back to the same level of prediction error and makes it easier for interpretation.

D. R- Squared

• R Squared measures how much variability in dependent variable can be explained by the model. It is the square of the Correlation Coefficient(R) and that is why it is called R Square.
• R Square is calculated by the sum of squared of prediction error divided by the total sum of the square which replaces the calculated prediction with mean. R Square value is between 0 to 1 and a bigger value indicates a better fit between prediction and actual value.

E. Adjusted R-Squared

• R Square is a good measure to determine how well the model fits the dependent variables. However, it does not take into consideration of overfitting problem. If your regression model has many independent variables, because the model is too complicated, it may fit very well to the training data but performs badly for testing data. That is why Adjusted R Square is introduced because it will penalize additional independent variables added to the model and adjust the metric to prevent overfitting issues